Latent-Domain Predictive Neural Speech Coding

Neural audio/speech coding has recently demonstrated its capability to deliver high quality at much lower bitrates than traditional methods. However, existing neural codecs employ either acoustic features or learned blind with a convolutional network for encoding, by which there are still temporal redundancies within encoded features. This paper introduces latent-domain predictive into the VQ-VAE framework fully remove such and proposes TF-Codec low-latency speech in an end-to-end manner. Specifically, extracted conditioned on prediction from past quantized latent frames so that correlations further removed. Moreover, we introduce learnable compression time-frequency input adaptively adjust attention paid main frequencies details different bitrates. A differentiable vector quantization scheme based distance-to-soft mapping Gumbel-Softmax is proposed better model distributions rate constraint. Subjective results multilingual datasets show that, low latency, 1 kbps achieves significantly Opus 9 kbps, 3 outperforms both EVS 9.6 12 kbps. Numerous studies conducted demonstrate effectiveness of these techniques.